System and methods of visualizing an environment

ABSTRACT

Exemplary embodiments described herein include methods of systems for visualization of test cell environments. Exemplary embodiments may include a virtual presence system and method of providing visualization that displays and permits virtual interaction with three-dimensional (3-D) data sets. Exemplary embodiments permit visualization through digital reality, such as Virtual Reality (VR), Augmented Reality (AR), and other display solutions.

PRIORITY

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/798,951, filed Jan. 30, 2019, which incorporated by referenceherein in its entirety.

BACKGROUND

Conventional monitoring systems include one or more cameras and/or othersensors that receive individual data feeds and present the informationto a user individually from the segregated feeds of each camera orsensor. For example, a user may capture one or more video images of anobject. The data feeds may be recorded for play back at a later time ormay be displayed to a user in real time. Conventional systems mayinclude user interface dashboards for simultaneously or selectivelydisplaying one or more of the data feeds from the one or more cameras orsensors. However, a user typically has to select which feed they areinterested in and manipulate the playback. Conventional systems displaythe playback on conventional two dimensional screens. Information oralarms may be set or based on the individual sensor to draw attention toa critical condition. However, the information is presented in isolationwithout a comprehensive correlation to the original physical objectbeing monitored or the physical environment.

To improve visualization of an area, a simple “lots of cameras with areally fast network” solution may be considered. However, such asolution will not suffice to cover an entire desired area whilesimultaneously delivering uncompressed video feeds with the desiredresolution. To achieve the desired resolution a large number of cameraswould be required that would overload practical system bandwidthlimitations.

In addition, the conventional feedback on conventional two dimensionaldisplays does not provide the same information as the three dimensionalenvironment provides. Details can be lost when viewing a threedimensional environment on a two dimensional screen.

BRIEF SUMMARY

Exemplary embodiments of the monitoring system described herein mayprovide comprehensive, three-dimensional (3D) visualization. Exemplaryembodiments described herein include a virtual presence system.

Exemplary embodiments may include a presence system and method ofproviding visualization that displays and permits virtual interactionwith three-dimensional (3-D) data sets. Exemplary embodiments permitvisualization through Virtual Reality (VR) and Augmented Reality (AR)solutions while preserving temporal and spatial registration. Althoughdisplay options are described herein in terms of AR/VR, other displayoptions are also included herein, including, without limitation, flatscreen approximations of the three dimensional rendering viewable on aflat screen or in augmented or virtual reality.

Exemplary embodiments may include devices for receiving data including,for example, imaging systems, temperature sensing, optical imaging invarious wavelengths, electrical sensor, mechanical sensors, other datasources, and combinations thereof.

Exemplary embodiments include a system of receiving data about an objectof interest. Exemplary embodiments are configured to superimpose thereceived data onto a three dimensional virtual object, where the threedimensional virtual object is a representation of the physical object ofinterest. In an exemplary embodiment, the system is configured toreceive information regarding the physical object. The system componentsmay be positioned in a known location and orientation relative to thephysical object such that data received from the system components maybe overlaid onto the virtual representation of physical test object.Exemplary embodiments of the system may be used to render, for example,collective video feeds into a realistic virtual reality environment.

Exemplary embodiments of the system may have any combination ofsegments, including, for example, a viewer system segment, a sensor nodesegment, and a network segment. The viewer system segment may includethe human to machine interface, such as a display system. The sensornode segment may include any combination of data collection nodes,system master timing and/or synchronization, and processing and/orstorage tasks local to the system. The network segment may includephysical data traffic infrastructure including, for example, switches,routers, cabling, etc. In an exemplary embodiment, the system mayinclude process piece in which the system is calibrated and initiated.Calibration may include setting and configuring sensor nodes and/orphysical mapping of the sensor nodes to the facility hardware, datafeeds, and observed object. Exemplary embodiments of the process piecemay align sensor nodes such that baseline three dimensional (or othervisualization from the viewer system segment) representation of theobserved object is aligned.

DRAWINGS

FIGS. 1A-1B illustrate exemplary environments and objects underobservation that may benefit from embodiments described herein.

FIG. 2 illustrates an exemplary method according to embodimentsdescribed herein.

FIG. 3 illustrates an exemplary high level system diagram of a test cellpresence system according to embodiments described herein.

FIG. 4 illustrates an exemplary component system diagram of a test cellpresence system according to embodiments described herein.

FIGS. 5A-5D illustrate exemplary visualizations from the test cellpresence system according to embodiments described herein.

FIG. 6 illustrates an exemplary test environment to illustrate themethods visualization according to embodiments described herein.

FIGS. 7A-7C illustrate exemplary data feeds from the test environment ofFIG. 6.

FIG. 8 illustrates an exemplary virtual representation of the testenvironment of FIG. 6 for visualization in three dimensions according toembodiments described herein.

FIG. 9 illustrates an exemplary system architecture according toembodiments described herein.

FIGS. 10A-10C illustrate exemplary dimensional mesh virtualrepresentations to illustrate exemplary embodiments of systems andmethods for generating a virtual object.

DETAILED DESCRIPTION

In the following description of preferred embodiments, reference is madeto the accompanying drawings which form a part hereof, and in which itis shown by way of illustration specific embodiments in which theinvention can be practiced. It is to be understood that otherembodiments can be used and structural changes can be made withoutdeparting from the scope of the embodiments of this invention.

System/Method Explanation—Visualization Based on Image Overlay

Exemplary embodiments may include Virtual Reality (VR) and AugmentedReality (AR) solutions that display and interact with three dimensionaldata sets while preserving temporal and spatial registration. Althoughdisplay options are described herein in terms of virtual reality, otherdisplay options are also included herein, including, without limitation,flat screen approximations of the three dimensional rendering viewablein augmented reality. Exemplary embodiments may include devices forreceiving data including, for example, imaging systems, temperaturesensing, optical imaging in various wavelengths, electrical sensor,mechanical sensors, and combinations thereof. Exemplary embodimentsdescribed herein may be hardware-agnostic and not tied to a specific VRor AR product and/or brand, allowing the customer to leverageappropriate VR/AR technology evolutions as they materialize.

Exemplary embodiments include a system of receiving data about an object(including an environment or multiple objects) under observation.Exemplary embodiments are configured to superimpose the received dataonto a three dimensional virtual object, where the three dimensionalvirtual object is a representation of the physical object.

FIGS. 1A-1B illustrate exemplary applications in which embodiments ofthe system and method for visualization described herein may be used.FIG. 1A illustrates a medical procedure in which a medical professionalis performing the medical procedure on a patient. The procedure may becaptured through one or more cameras and replayed or observed in realtime. Such recording or display may be used for training purposes or formaintaining a record of the procedure or other purpose. Conventionally,the captured visual data feeds would be replayed in isolation, such asthrough display on one or more two dimensional monitors. However, suchtwo dimensional segregated presentation of the information may notprovide the same experience for a viewer as the opportunity to observethe actual procedure in three dimensions. Information about relativepositions may be lost on a viewer merely watching individual feeds fromone or more camera feeds. FIG. 1B illustrates an exemplary object thatmay be observed during operation, such as in a facility or industrialapplication. One or more cameras may be used to monitor, record, observeor combinations thereof the object during use. The system may capture,display, or record the information for observation in real time or at alater time. Such applications may permit remote viewing or monitoring,facility monitoring, infrastructure monitoring, quality assurance, faultdetection, forensic assessment for recovery or isolation during a faultoccurrence, or in damage assessments after a fault occurrence.

Conventionally, each feed may be observed and/or recorded individually.A user thereafter observes the various individual feeds. Although one ormore feeds may be visually present and visible to a user (such as a userdisplaying two separate video feeds simultaneously), there isconventionally not a convenient way to integrate the information for abetter or complete understanding of the object under observation.

Exemplary embodiments described herein provide a system and methods forproviding an integrated view of an object under observation includinginformation from one or more sources. FIG. 2 illustrates a flow diagramfor methods of visualizing data of an object under observation byoverlaying received data onto a virtual object corresponding to thephysical object. The exemplary method includes receiving informationabout the physical object, providing a virtual object corresponding tothe physical object, receiving information from one or more sources, andoverlaying the received information onto the virtual object to providean integrated view of the test object.

As represented at step 202 of FIG. 2, an exemplary method according toembodiments described herein include providing a physical environmentincluding an object for observation. The physical object may be anyphysical object, group of objects, or environment for observation.Observation is intended to be inclusive of any objective includingvisual observation as well as specific monitoring, physical testing(such as run time monitoring/testing or environmentalmonitoring/testing). Run time and environmental testing may includeoperating an object in different environments, including dynamic(changing) environments of temperature, pressure, humidity, vibration,acceleration, movement, etc. Observation may also include any observableattribute of an object, not necessarily limited to visual observations.For example, observations may be through sensed information, such astemperature, speed, object input (such as power, current, etc.), objectoutput (such as exhaust, power, current, light, heat, etc.), and anycombination thereof.

As represented at step 204, the exemplary method includes providing apresence system according to embodiments described herein. The presencesystem may include one or more data sources to observe the physicalobject of step 202. As described above, the observations may be throughany combination of attributes. In an exemplary embodiment, the test cellpresence system comprises one or more cameras. The exemplary cameras maybe in one or more bandwidths, such as for visual observation indifferent spectrums, including without limitation, visual, infrared(IR), ultra violet (UV), or other frequency such as for night vision,heat detection, etc. The one or more data sources may be any combinationof sensors. In an exemplary embodiment, the sensor may be, for example,IR, vibration, UV, visual, audial, temperature, speed, current,composition, etc.

At step 206, the method includes providing a virtual representation ofthe physical object. In an exemplary embodiment, to create the overlayof the data onto the virtual object, exemplary embodiments may includecreating an accurate three-dimensional representation of the physicalsetup including the object for observation and/or test hardwarecomponents. Exemplary embodiments may use modeling or other rendering tocreate a virtual representation of an exemplary physical environmentincluding the object under observation. In an exemplary embodiment, testready computer aided design (CAD) models may be used as a basis for thevirtual object. Any method for creating a virtual representation of thephysical environment and/or test object are within the scope of theinstant application. For example, methods for generating a virtualrepresentation from a physical object or environment may include laserscan photometric scan, or other detector, system, or method ofgenerating a three-dimensional rendering. Exemplary methods to createaccurate three dimensional renders of the object and/or test cellhardware and/or environment may include any combination of steps,including, without limitation, CAD modeling of the object and/orcomponent parts, object detection and rendering through one or moresensors, image recognition, and combinations thereof.

At step 208, the physical environment including the object forobservation (test object) may be mapped to the virtual object. Thesystem may therefore be calibrated and/or initialized such that thephysical mapping of the facility hardware, data feeds, and the observedobject correspond to and properly align when overlaid onto the virtualrepresentation. In this step, the system components may be positioned ina known location and orientation relative to the physical object suchthat data received from the system components may be overlaid onto thevirtual representation of physical object. Other calibration systems andmethods may also be used. For example, manual alignment may be used toalign the visual feedback to the overlaid virtual object. The manualalignment may be performed in physical positioning of the sensors andcamera, in electronic or software manipulation of the alignment of theoverlay to the virtual objects and combinations thereof. In an exemplaryembodiment, the system may be automated to detected a position of thesensors and determine a corresponding alignment for the sensor feed foroverlaying on the virtual representation. For example, image recognitiontechniques may be used to identify a position on a camera feed tocorrespond with a position on the virtual representation. In anexemplary embodiment, the system may integrate one or more sensors intoa data feed such that the data feed is in a predetermined locationrelative to a sensor for determining its position relative to the testobject or other known environmental position. The data feed maytherefore be able to self-locate and its data feed overlaid on thevirtual object automatically. Exemplary embodiments may includecombinations of automatic and manual calibrations. For example, thesystem may be manually calibrated to a set up orientation. However,during a test procedure or observation sensors may be permitted to move,rotated, or otherwise reposition. The repositioning of the sensors maybe performed through command signals to mechanical/electrical componentssuch that the repositioning is by a known amount. The system maythereafter automatically recalibrate based on the known changes to thesystem configuration. The system may also initially automaticallycalibrate, but may permit manual adjustments to improve or correctautomatic determinations.

As represented by step 210, the method includes receiving informationregarding the physical environment, including, for example, the physicalobject under observation. The system may be configured to receiveinformation from any of the described data sources or other source.Information may come from data of the one or more data sources,including cameras, sensors, etc. The information may come from senseddata, analyzed data, received data, data input, etc.

At step 212, the method may include manipulating the received data insome way. The system may be configured to aggregate the data sources forrepresentation on the virtual object. The system may aggregate the datasources by aligning the data sources. For example, the data may beaggregated by synchronizing the feeds in time. The data may beaggregated by aligning the data relative to a corresponding relativephysical location. For example, data may be overlaid, duplicated,filtered, and combinations thereof for portions of data sources thatoverlap. In an exemplary embodiment, one or more data sources mayprovide a panoramic view of a test object, but may include overlappingareas between data sources. Aggregating the information may includealigning the feeds, and filtering overlapping data. Filtering may be byaveraging information, removing information, etc.

Exemplary embodiments may also include the addition of dynamic datasources. For example, a user input through the user interface maygenerate data that can be appended to a data source or data stream orvisual representation or recreation. For example, a user may look at thevirtual representation using the user interface as described herein. Theuser may provide an input to the system, such as through an electroniccontroller (for example, a button push or movement queue). The userinput may provide a tag or other input to the system that can be storedwith the data for recreation or review in real time or replay. The tagmay permit a user to enter in additional information, such as notes, orobservation queues, or may simply identify points of observation topermit searching, training, record keeping, or other data manipulationat a later time.

The system may perform other data analysis or synthesis. For example,the system may be configured to reduce a fidelity of one or more datasources to improve band width transmission. Fidelity may be reducedbased on level of observation. For example, the lower fidelity (lessdata) may correspond to more distant points of view or larger areas ofobservation, while a higher fidelity (more data) may be provided formore specific areas of observation. The system may be configured toidentify or receive areas of interest in which higher areas of fidelityare desired and/or lower areas of interest, either of which may indicatethe converse to the system. The fidelity may also be set based onreceived information, historical information, rates of change, etc. Ifthe received information is within normal tolerances or a set tolerance,or within a given rate of change relative to a historical value, thesystem may reduce the fidelity as the received information. If thereceived information is changing, close to or within or outside of arange of observation or predefined or determined range, or othercriteria, the system may be configured to receive or capture a higherfidelity. Fidelity may be, for example, a sampling rate of a givensensor or density of information such as in higher resolution. Exemplaryembodiments may also perform analysis of one or more data sources orfeeds for event detection. The system may be configured to adjust afidelity of information based on the detection of an adverse or knownevent or other system criteria.

At step 214, the method may include storing the information. The systemmay be configured to store any combination of information. For example,the system may store the raw data feeds from the one or more datasources. The system may store the aggregated data sources. The systemmay store any analyzed, synthesized, or any combination of manipulateddata. The system may also store the visualization of step 216.

Exemplary embodiments of the system may be used to render, for example,collective video feeds into a realistic virtual reality environment. Themethod, at step 216, may include rendering information onto the virtualrepresentation of the physical object. The visualization may be throughany digital interface, such as a monitor, screen, virtual realitydisplay, augmented reality display, etc. The visualization may bethrough augmented reality and/or virtual reality or other threedimensional digital display (referred to collectively herein as digitalreality). In this instance, the virtual representation of the physicalobject may be rendered and displayed in three dimensions. Theinformation corresponding to the physical environment may be overlaidonto the virtual representation such that the received information isdepicted visually directly over, onto, or proximate the virtual object.The user may therefore receive an approximation of the physical objectduring the observation in virtual space as if observing directly inphysical space. The representation and/or overlay may alter the visualof the representation for the viewer such that it is not the same as adirect observation of a physical object. This may be, for example when atemperature or camera detecting in a non-visual spectrum is used andoverlaid such that the virtually rendered object with informationoverlaid thereon may be represented in color corresponding totemperature, similar to a three dimensional heat map.

Test Cell Presence System

Exemplary embodiments described herein include systems and methods forproviding a virtual presence system in which an object may be observed.The observation may include additional information beyond (or inaddition to) visual inspection, such as through different frequencies,temperature, or other sensor information, and/or may include remoteinspection by a viewer removed from the test location or facility.

FIG. 3 illustrates an exemplary block representation of a virtualpresence system according to embodiments described herein. Exemplaryembodiments of a virtual presence system 300 may include any combinationof segments, including, without limitation, a view system segment 304, asensor node segment 302, and a network segment 306. The method may alsoinclude a processing segment 302A. Exemplary embodiments are describedherein in terms of different segments for example and explanation only.The system does not require any specific integration or segregation ofsegments. For example, any combination of components may be used aswould be apparent to a person of skill in the art.

View System Segment

In an exemplary embodiment, the view system segment may include a userinterface for displaying the results described herein. The view systemsegment 304 may include any combination of displays, includinginteraction stations 312 that permit user input and machine outputincluding any digital display (augmented reality, virtual reality, 2-Dscreen, hologram, etc.). The user interface may be through a display orhuman machine interface. An exemplary embodiment of the display includesa virtual reality or augmented reality display/user interface. Otheruser interfaces may include digital displays such as 2-D screens,projectors, holograms, or other visual display system. Exemplaryembodiments of the system are configured to display a virtual renderingof the object under observation with or without an environment aroundthe object. The object may be the environment itself, and does notrequire a specific component for observation. The system is configuredto display virtual representations of information about the physicalenvironment including the physical object overlaid onto, positionedadjacent, or otherwise in relation to the virtual rendering of theobject. In an exemplary embodiment, the representations of informationis a camera feed conformed about the virtual rendering of the objectsuch that the display of the representation of information with thevirtual object is a recreated three dimensional view corresponding tothe physical object under observation as seen by one or more sensors,including one or more cameras. Other information may be overlaid ordisplayed on the virtual rendering of the object, such as, for example,color coded areas, call outs, text, or other display of information inrelation to the virtual rendering of the test object corresponding tothe information of the physical object.

Sensor Node Segment

The sensor node segment 302 of the system 300 may include anycombination of sensors, controls, processing, or other components 308for collecting the information for display. An exemplary embodiment ofthe sensor node segment is configured to receive data from the physicalenvironment and/or physical object. The nodes may include any sensor,such as a camera, thermal detector, etc. The sensor node segment mayalso include components for system master timing and/or synchronization,one or more processors, and one or more memory for storing tasks and/ordata associated with the system, and/or controlling the one or moresensors or other sensor node segment components. In an exemplaryembodiment, the sensor node segment or one or more components of thesensor node segment may be positioned within an observation environmentboundary. The observation environment boundary may segregate theobservation environment from a remainder of the environment and/or oneor more users. The observation environment boundary may be used tocontain an environment, such that temperature, pressure, humidity, andother environment factors may be controls, as well as contain chemicals,exhaust, heat, or other hazardous or unhealthy conditions from humanobservers.

In an exemplary embodiment of methods using embodiments of the virtualpresence system, the system 300 may be calibrated in a processingsegment 302A. For example, the processing segment 302A may include thecalibration of sensor nodes from the sensor nodes segment 302, includingillumination and camera performance parameters, and physical mapping ofsensor nodes to the physical object and/or facility hardware and datafeeds. The alignment of sensor nodes baselines the 3D visualization andmay complete the system initialization. As described herein, thecalibration of the virtual presence system may be manual, automated or acombination thereof.

Network Segment

The network segment 306 may include one or more components such asnetwork hardware, timing, communication, etc. 314. An exemplaryembodiment of the network segment includes methods and components forcommunication between different components of the system and/or to orfrom the test environment. For example, the physical data trafficinfrastructure including switches, routers, and cabling that connectsthe system components.

System Architecture

FIG. 4 illustrates an exemplary test system architecture according toembodiments described herein for the virtual presence system.

The virtual presence system 400 includes a location for the physicalobject 402. The physical object 402 is any physical object forobservation and/or testing in the environment. The observationenvironment is defined by the observation environment boundary. Theobservation environment boundary may be a physical boundary orseparation or may simply be imposed by the field of view or detection ofthe one or more sensors. As described herein, the test environmentboundary permits the delineation, separation, and/or control of theobservation environment including the physical object under observation.The observation environment boundary may be sealed, such as to controlpressure environments, may be sealed and/or vented to contain hazardousmaterials, or may include other structures, components, and features asthe environmental needs dictate for performing the desired observationof the physical object.

Exemplary embodiments include sensors 406 and control systems 408 withinthe observation environment for receiving and transmitting data aboutthe physical object 402. The sensors may be any combination of datareceiving inputs, such as different video-source sensors 404 (e.g.cameras) utilized to provide video coverage across different wavebands.Any combination of sensor types, quantities, qualities, locations, etc.may be used within the scope of the present disclosure. Differentsensors and cameras are illustrated in FIG. 4 as cameras C1-C4 andsensors S1-S4. As used herein a camera is a type of sensor. Sensorhardware may be based upon a specific test environment and designed orconfigured for specific imaging requirements and may vary byinstallation or test. Exemplary control systems 408 may be ruggedizeddepending on the local test unit environment and hardware used for thecommand/control unit. The system may provide other components based onthe test environment, such as, for example shock isolation. The controlsystem 408 may manage video traffic and providing accurate timing acrosssensor nodes. The control system 408 may be positioned based upon cablelengths, and environmental considerations. The control system 408 mayinclude memory to provide local data storage.

Exemplary embodiments of the test cell presence system 400 may include adata aggregation hub 410. The data aggregation hub may include one ormore processor and one or more memory and other components for managingthe synchronization, video routing, command traffic, or other featuresof the network segment described herein. The aggregation hub 410 mayreceive the data feeds from the sensors within the test environment. Thedata aggregation hub 410 may also be configured to perform any of thedata aggregation, analysis, filtering, synthesizing or othermodification of the raw data from the sensors from the test environment.The data aggregation hub may be proximate the test environment or may beremote therefrom.

Exemplary embodiments of the system may include a viewer system segmentincluding user visual displays. Any 2-D screen 412 or user displaydevice may also be used. Alternatively, or in addition thereto, anydigital (either virtual, augmented, or holographic) reality system 414may be used. In an exemplary embodiment, the digital reality display mayuse “inside-out” tracking with all tracking hardware present on aheadset. Other tracking and control inputs may also be used. Forexample, a controller, such as a handheld remote may be used. Theexemplary tracking and control components may be used to alter the viewof the digital reality by changing perspective, zooming, changingdisplay information/inputs, or combinations thereof. Exemplaryembodiments may reduce the connections needed between the headset andthe rest of the system.

Exemplary Displays

FIG. 4 illustrates exemplary virtual representations of a physicalobject as viewed through a digital display 414A, 414B, 412 according toembodiments described herein. The exemplary representations of thevirtual reality displays 414A, 414B illustrate the same object withdifferent information overlaid on the virtual object to provide examplesof how information can be provided to a user through a three dimensionalvirtual representation of the physical object.

FIG. 5A-5D illustrate an exemplary display options in which informationis display to a user in combination with the virtual representation ofthe physical object. In an exemplary embodiment, non-imaging data may beprovided with the virtual representation of the test object on a displayas a pop up. Exemplary embodiments of the system and method areconfigured to receive different sources and types of data. The receiveddata may not include visual or imaging data that can be grafted onto theshape or virtual model of the physical object for a direct overlay ofthe data onto the virtual representation of the physical object.However, this non-conforming information may be displayed in other ways.As illustrated in FIG. 5A-5D, the information may display on a pop up orinformation display window displaying the data approximate to or with avirtual object indicating the source of the information. Other displayoptions may also be used, such as providing other virtual objectoverlays. For example, color coding or symbol corresponding orrepresenting the displayed data may be used as an overlay of the virtualrepresentation of the physical test object. As illustrated in FIG. 5A,if a temperature sensor is determined to be out of range, the locationof the temperature sensor on the virtual representation of the physicalobject may change color or a symbol (illustrated as a star in FIG. 5A)may be used to draw attention to that location of the virtualrepresentation of the physical object. Different temperature color codesmay be used to correspond to or indicate different things, such as inrange, out of range, high, low, or approximate temperature range, etc.Also as illustrated in FIG. 5A, text information or other informationfrom a data source may be provided as an overlay positioned in proximityto the virtual representation of the test object corresponding to thesource of data represented in the overlay. As illustrated thetemperature associated with the symbol displayed on the virtualrepresentation is displayed to a user.

FIGS. 5A-5D illustrate exemplary demonstrations of a rendered samplevirtual environment to demonstrate the notional system user experience.Temperature sensors and other non-imaging data sources could displaytheir status via colored or symbolic indicators on the VR model (FIG.5A). Detailed information may appear when the data feed node is “lookedat” by the user. Navigation in the virtual environment may control,engage, interact with, and/or view different portions of the system. Forexample, the system may be configured to detect head movements, whichmay be used to control the short distance travel and precisionpositioning of the virtual display environment. Exemplary embodimentsmay also include hand controllers or other inputs that may be used totrigger long distance movement within the virtual display environment.For example, the system may include a hand held controller that mayinclude a joy stick, buttons, toggle, or other controller(s). In anexemplary embodiment, the user may select to “move to” a given targetvia the thumb-stick command or other input. Left, or right movement ofthe thumb-stick may be used to adjust the view's rotation, while reversemovements may be activated by pushing backwards on the thumb-stick.Exemplary embodiments, may suspend or otherwise manipulate the displayof the virtual environment, such as through fade in/out, blinking,screen freeze, or other transition to minimize user disorientationduring teleporting or rotation from one view or viewing area of thevirtual environment to another. As a navigation aid, the system may beconfigured to receive or define locations for rapid repositioning. Thehand controller may superimpose representative user interface controlsand information display as illustrated herein. The interface (asillustrated in FIG. 5D) may be normally hidden within the virtualdisplay environment and may be called up via an input, such as a menubutton or head motion input, and then operated with a controller, input,or gesture recognition.

Exemplary embodiments described herein include different implementationsfor the system, ranging from a simple distribution of a small number ofcameras that are approximately located in a less-than-detailed threedimensional model to a high-fidelity rendering of the physical objectwith live video feeds from numerous precisely aligned and calibratedcameras draped seamlessly onto an accurate three dimensional model ofthe physical object.

Exemplary embodiments permit the operator(s) to set up fixed “virtual”display feeds that deliver information to a standard two dimensionaldisplay. This enables other personnel to view selected imagery feedswithout virtual reality. The system may be used to render the user'svirtual reality point of view to a two dimensional display, and/orpermit the user of the two dimensional display to rotate or navigate thevirtual object through the two dimensional interface. Thetwo-dimensional display may also provide information about the threedimensional user perspective. As seen in FIG. 5C, an exemplary virtualrepresentation of the test object may be displayed with the perspectiveposition of the viewer through a virtual reality or other threedimensional display system is indicated to represent the focus,perspective, and view of the three dimensional viewer to the twodimensional viewer. As illustrated, the headset/lenses 502 of the threedimensional viewer are represented on the virtual representation of thephysical environment to indicate position and direction of the threedimensional viewer.

Exemplary embodiments may permit multiple users to engage with thesystem simultaneously through any combination of user interfacealternatives described herein. For example, one or more users mayexperience the digital reality as well as one or more other users mayexperience through two-dimensional displays or even single dimensiondata feeds. The system may be configured to permit users to control andinteract through the user interface either independently, such that eachuser can manipulate their personal view and receiving corresponding datafeeds, or collaboratively such that views may be shared or manipulatedcollectively, or any combination thereof.

Recall and Storage

Exemplary embodiments of the system described herein may include storagefor retaining raw data feeds, analyzed data feeds, visual feeds, and anycombination thereof. For example, the system may capture, store, andreplay the user interaction with the system during a use session suchthat the visual experience of a user may be captured and replayed. Thesystem may also record raw feeds such as from sensors such that specificinformation may be replayed as desired. The system may also beconfigured to further analyze, manipulate, or otherwise handle any ofthe retained data in order to replay any combination of information orgenerate new information or displays. In an exemplary embodiment, thevirtual representation and/or one or more data feeds may be stored suchthat a user may recreate the three dimensional display on demand. Theuser may thereafter interact with the system, such as through movementdetection or other user input, to manipulate the user displaydynamically during the replay session.

In an exemplary embodiment, the system may be configured to record oneor more data feeds from one or more sensors. For example, referring backto FIG. 1A, a medical procedure may be captured and recorded from one ormore cameras. The system may also include one or more systems or methodsfor generating a virtual dimensional mesh in which to overlay the camerafeeds. The system may permit remote observation and/or replay of theprocedure through a three-dimensional display system. The observer,through the system described herein, may interact with the display suchthat the medical procedure can be observed from different directions,replayed, paused, zoomed in or out, or otherwise manipulated. One ormore users may simultaneously or separately replay or interact with thesystem such that the display experience can be shared or can remainindependent and separate. The system and methods described herein may,for example, be used for training after a procedure is complete. Thetraining session may use recordings of an actual procedure, but permit atrainer to pause the procedure or provide specific perspective useful tothe training session without interfering with the procedure itself. Asanother example, referring to FIG. 1B, an industrial process oroperation may be observed and/or monitored. The remote user may beseparate from the environment of the object under observation for anyreason, such as health concerns, physical location separation, etc. Thesystem may permit the three dimensional observation of the object toretain perspective of the object under observation that may be lost ifobserved on a conventional two dimensional display.

The system may also be configured to take in data feeds from othersources. Referring back to FIG. 1A, the inserted medical device may havea tracker or other location detecting sensor at an end of the device.The system may be configured to provide a three dimensional image of thepatient undergoing the procedure, and may provide an additional virtualobject representation in the three dimensional display representing thelocation of the medical device as determined by the location detectingsensor and may position the virtual object representation relative tothe virtual patient based on the relative position of the location thephysical device to the physical patient. The location of an object isprovided as merely exemplary, any additional information may be receivedand displayed in the system. Referring back to FIG. 1B, the system maypermit a user to observe the object and overlay additional information,such as a visual feed, temperature, pressure, etc. data. The additionalinformation may be represented as text overlaid in a positionapproximate or correlated to the location on the virtual object. Theadditional information may also be overlaid on the dimensional mesh orvirtual representation, such as by a color coding or symbolic scheme.

FIGS. 6-8 illustrates an exemplary system reconstruction to illustratethe system and concepts described herein. A box 602 is chosen as a testobject for observation. The box is virtually modeled and a threedimensional mesh model is used to render the video feeds from threecamera sources C1, C2, C3 onto the virtual representation of thephysical object. As illustrated in FIG. 6, the test environment includesthe target object 602 under observation and three cameras C1, C2, C3. Asillustrated, a bust figuring is used to illustrate an imagingobstruction in one camera feed, C1. Therefore, since the system has noknowledge that the figure is an obstruction, the system renders the bustonto the model. FIG. 7A illustrates the image received from camera C1;FIG. 7B illustrates the camera feed from camera C2; and FIG. 7Cillustrates the feed from camera C3. As illustrated, the bust figureappears in the image of camera C1 in front of the target object 602.FIG. 8 illustrates the virtual representation of the physical objectwith the information from the camera feeds superimposed onto the virtualmodel. As illustrated, the image of the bust is integrated onto the sideof the virtual representation of the cube as the system is unaware thatthe feed is obstructed and does not correspond to the model representingthe physical object.

In an exemplary embodiment, the system may include depth sensors as oneor more components of the virtual presence system. The depth sensors maybe used to generate the three dimensional mesh or structure for modelingthe virtual representation of the physical environment. An exemplary,therefore, may include a system and method of providing or receivingdepth sensor outputs for use in an exemplary embodiment to create athree dimensional mesh for use in the virtual object overlay. In anexemplary embodiment, a three dimensional rendering method may includethe user of a depth sensor, either in combination or separate from thecamera or video feed. In an exemplary embodiment, a combined colorcamera depth sensor is used. Exemplary embodiments can be used to createa three dimensional mesh for the perspective of the camera.

FIGS. 10A-10C illustrate exemplary renderings of a virtual object withimage overlay based on a dimensional mesh created from one or more depthsensors as described herein. FIG. 10A illustrates the front, and FIG.10B illustrates a side, and FIG. 10C illustrates a top view of anexemplary depth sensor output for use in an exemplary embodiment tocreate a three dimensional mesh for use in the virtual object overlay.In an exemplary embodiment, a three dimensional rendering method mayinclude the user of a depth sensor, either in combination or separatefrom the camera or video feed. In an exemplary embodiment, a combinedcolor camera depth sensor is used. Exemplary embodiments can be used tocreate a three dimensional mesh for the perspective of the camera.

In an exemplary embodiment, the virtual object or dimensional mesh foroverlaying information or rendering a video display for threedimensional display may be dynamic such that the dimensional mesh maychange and correspond to the physical environment in real time orsemi-real time. To reduce processing or band width or improve fidelity,the system may be configured to update the dimensional mesh for aportion of the virtual representation while maintaining other portionsstatic. For example, if the observation is on a surgery or industrialoperation in which only a portion of the object of observation is movingor changing, only that portion of the virtual representation needs to beupdated. Accordingly, the system may be configured to automaticallydetect changes and update portions of the virtual representationaccordingly and/or may be programed to update the virtual representationfor an identified region of the virtual representation.

Exemplary embodiments may be configured to resolve small objects (forexample, ˜0.05 inches or less). Exemplary embodiments of the system andmethods described herein may allow the user to “walk around” in digitalreality and monitor critical joints, items and connection points. Thesystem may deliver multi-spectral sensing capability (visible andinfrared wavebands, as examples) with continuous, real-time 3D videofeeds. The system architecture may support integration of active viewingof other data sources (temperatures, pressures, data feeds, etc.). As asystem, exemplary embodiments permit faster visualization and a morecomprehensive understanding of the operational environment, helpingdetect minor issues before they grow into major problems.

Exemplary embodiments may include system architectures that may considerboth the large amounts of real-time data required for rendering the testcell presence into virtual or augmented reality and the practicallimitations of today's state-of-the-art computers. Exemplary embodimentsof a virtual presence system implementation may include sensible cameraselection, appropriate network design, intelligent bandwidth management,and practical considerations about the physical environment coveragerequirements. Large-scale, high-resolution viewing of the physicalenvironment unit may include a form of video compression, or a methodfor video feed switching implemented as a “Level of Detail” viewingcapability. “Level of Detail” may automatically (or manually) reduce theresolution of the camera field into digital reality or for displaydepending on the virtual distance between the viewer and camera andactual resolution of the display. “Level of Detail” may adjust theresolution or other fidelity (sampling rate, etc.) displayed in digitalreality or other display methods depending on a virtual distance betweena virtual viewing perspective and the virtual representation of the testobject. For example, if a user through the digital reality interfacemoves closer to the virtual representation of the test object, thefidelity or resolution of the display may increase, while the fidelityor resolution may be reduced as the viewer digitally moves further awayfrom the virtual representation of the test object. These methods may beincorporated to preserve transmission bandwidth.

Exemplary embodiments may use any combination of hundreds of potentialcameras. Any combination of cameras, sensors, and data sources may beused in any combination. Therefore, there may be a single camera or anynumber of multiple cameras, sensors, or other data feeds or sources. Thecameras vary by waveband, image type, focal plane size, pixel pitch,frame rate, data output format, interface, environmental performancerange, and other parameters. In an exemplary embodiment, the systemincludes hard-mounted sensors with fixed focal length lenses. The fixedlocation and focal length of the cameras may provide for easiercalibration and mesh overlay of the received data on the renderedvirtual object. The system may also use variable locations and/or focallengths in which the system may be manually or automaticallyrecalibrated according to embodiments described herein. In an exemplaryembodiment, highly stabilized and steerable custom imaging systems maybe used that provide accurate and repeatable positioning.

Calibration of exemplary embodiments described herein may include whitebalance, and other performance parameters. Calibration may include thephysical three dimensional mapping of the physical system components,physical object under observation, and the relative alignment of thesenor nodes to the three dimensional map. A calibration process permitssensor node alignment and permits the proper generation of threedimensional imagery from the two dimensional video feeds. Calibrationmay be used to establish various intrinsic and extrinsic parameters ofthe respective sensor nodes and may record them as part of aninitialization process. Intrinsic parameters (lens focal length, camerapixel pitch, etc.) remain fixed throughout the lifetime of the sensornode, while extensive parameters such as sensor node position andorientation may vary due to operational needs. The use of fixed systemreference points and rigid mounting techniques helps minimizerecalibration burdens.

In an exemplary embodiment, the system may also include dynamic orcontrollable intrinsic parameters, such as camera position, orientation,focal length, etc. The system may be configured to detect a change in adynamic intrinsic parameter and recalibrate the system accordingly. Therecalibration may be automatic, manual, or a combination thereof. Thesystem may also include one or more identification sensors to assist incalibration. For example, the system may detect or determine a location,use visual or other data recognition to relate a data stream to thevirtual representation to permit calibration and data overlay to thevirtual representation.

Exemplary embodiments of the system including a multi-camera system canbenefit from using a master timing device and master clock. Exemplaryembodiments of a system architecture is illustrated in FIG. 9. Thesystem may include any combination of switch controllers 902 coupled toany combination of cameras C1-Cn. One embodiment may use IEEE 1558compliant cameras to simplify system level timing synchronization andsynchronizing all computers, cameras, and networking equipment in asystem. Other standards, protocols, components, methods, andcombinations thereof may also be used for timing, synchronization, oramalgamating data. A master clock can aid in determining and fixing anysources of latency that may occur. The system may include anycombination of data aggregation hub 912 or other analytics components asdescribed herein. The system may include any combination of digitaldisplays 904 for rendering the virtual object in conjunction withinformation from the one or more data sources. The system may beintegrated into a conventional or previous system architecture 910 andprotected through a fire wall 908 and have access to the system networkor internet 906.

An exemplary network may include a physical topology that supportsfuture increases in camera count and capability. The transfer of videofrom cameras to optional local data storage nodes may also be used tominimize sharing of links and allows direct calculation of the bandwidthand storage capacity requirements. Long-haul links, such as thosebetween the test area and display area, may use fiber. Remaining linksmay be copper, unless greater resolution is required.

Exemplary embodiments of system components of the system network mayinclude managed equipment, in that they have a dedicated managementinterface and maintain operational statistics. This visibility intonetwork behavior may be used to verify the configurations and expectedresults against real system operations.

Exemplary embodiments seek to minimize unmonitored choke points in whichexcessive flows of video data converge. The network structure may beused to increase flexibility in resource allocation and can expand toincorporate additional cameras and storage nodes on an as-needed basis.

Exemplary embodiments may use various configurations of videocompression or various compression techniques such as H.264, H.265, JPG,and JPG2000. These compression methods could reduce the amount ofbandwidth required by the network but introduce latency and require someform of processing power and may also reduce overall image fidelity.

Exemplary embodiments of the system described herein can be based insoftware and/or hardware. While some specific embodiments of theinvention have been shown the invention is not to be limited to theseembodiments. For example, most functions performed by electronichardware components may be duplicated by software emulation. Thus, asoftware program written to accomplish those same functions may emulatethe functionality of the hardware components in input-output circuitry.The invention is to be understood as not limited by the specificembodiments described herein, but only by scope of the appended claims.

Although embodiments of this invention have been described withreference to the accompanying drawings, it is to be noted that variouschanges and modifications will become apparent to those skilled in theart. Such changes and modifications are to be understood as beingincluded within the scope of embodiments of this invention as defined bythe appended claims.

1. A method of providing three-dimensional visualization, comprising:providing a virtual representation of a physical object; receivinginformation related to the physical object; visualizing the informationon the virtual representation.
 2. The method of claim 1, wherein thereceiving information related to the physical object comprises receivingone or more digital feeds from one or more sensors.
 3. The method ofclaim 2, wherein at least one of the one or more sensors comprises acamera.
 4. The method of claim 3, wherein the visualization of theinformation comprises overlaying a data stream from the camera over atleast a portion of the virtual representation of the physical object. 5.The method of claim 4, wherein the visualization of the informationcomprises displaying the virtual representation of the physical objectwith the overlaid data stream.
 6. The method of claim 5, furthercomprising calibrating the one or more sensors to the virtualrepresentation of the physical object.
 7. The method of claim 6, furthercomprising manipulating the data stream by reducing a resolution basedon a level of detail depending on a virtual distance between a virtualviewing perspective and the virtual representation of the physicalobject.
 8. The method of claim 7, further comprising receiving an inputfrom a user.
 9. The method of claim 6, wherein the providing the virtualrepresentation is generated through a control aided design model. 10.The method of claim 6, wherein the providing the virtual representationis through one or more sensors to generate the virtual representation.11. A virtual presence system, comprising: a plurality of camerasconfigured to receive information about a physical object, eachgenerating a data stream; a hub configured to receive the data streamsand aggregate the received data stream into an aggregate data stream; adigital reality display configured to display a virtual representationof the physical object with the aggregate data stream overlaid on thevirtual representation.
 12. The virtual presence system of claim 11,wherein the hub is configured to receive the virtual representation ofthe physical object as a computer aided design model.
 13. The virtualpresence system of claim 11, further comprising one or more sensorsconfigured to provide information to the system, and the system isconfigured to generate the virtual representation of the physicalobject.
 14. The virtual presence system of claim 11, further comprisingone or more sensors configured to calibrate the plurality of cameras tothe virtual representation of the physical object to align the overlayof the aggregate data stream on the virtual representation.